Avoiding the Comparative Fallacy in the Annotation of Learner Corpora

It is becoming more common to use corpora of second language learner data in order to support research on various second language acquisition (SLA) topics (e.g., Römer, 2009; Wulff et al., 2009), but there has been little use of corpus annotation. For many questions in SLA research, using a corpus is simple and in no need of annotation: one can search a corpus for specific words to find relevant examples. For example, if one wants to examine how modal verbs are used by L2 learners (cf., e.g., Aijmer, 2002), one can search for those specific lexical items (can, should, etc.) and analyze the output by hand. Consider a search for syntactic patterns, however, such as examining wh movement (e.g., Juffs, 2005; Wolfe-Quintero, 1992; Schachter, 1989). These types of questions require more linguistic abstraction (cf., e.g., Lüdeling, 2010). If we take the learner sentence (1), for example, what kind of search involving specific words addresses questions about the function of whom?1 If we search for all instances of whom in a corpus, we still have to determine whether this is a relative clause marker, whether this is subject or object extraction, or what the depth of embedding is; and then we need to do the same for that, which, or even other prepositional objects. We need the data marked with syntactic annotation.